Methods and System for Monitoring and Assessing Employee Moods

ABSTRACT

Methods and system for monitoring and assessing employee moods are disclosed. A proposed enterprise employee monitoring system includes surveillance cameras, a facial recognition module, an emotional analyzer module, and an employee database. The surveillance cameras capture image data including employee individuals within the enterprise. The facial recognition module identifies the individuals in the image data, and the emotional analyzer module determines an emotional state of the individuals based upon the image data. The employee database stores employee information and the emotional state information from the emotional analyzer module, based upon the identification performed by the facial recognition module.

RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Application No. 62/581,207, filed on Nov. 3, 2017, which isincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

Enterprises, such as private and public companies, municipal, state andfederal governmental agencies, and other entities, will often maintain anumber of disparate systems to facilitate their operations, track theirbusiness relationships, and maintain security. Employee ResourceManagement (ERM) systems are computer systems that allow the enterprisesto track, schedule, and pay their employees. Access control systems areprincipally concerned with physical security and the selective accessto, restriction of access to, and/or notification of access to theenterprises' buildings and secured parts of those buildings. Inaddition, other security systems are often employed by the enterprisesto round-out their security needs. A common example is a surveillancesystem.

The ERM systems store and manage many different types of informationassociated with employees. The ERM system might execute on a singlecomputer system or server, or across multiple computer systems andservers, or be implemented in a cloud-based computer system. Thedifferent types of information controlled and managed by the ERM systemsinclude biographic, including demographic, information, payroll andsalary information, job performance and attendance information, benefitsinformation, and training and compliance information, to list somecommon examples.

Modern ERM systems typically combine the functionality of multiplelegacy systems that had separately managed and stored the differenttypes of information associated with the employees. These legacy systemsmight have had separate payroll systems for the payroll and salaryinformation, human resources systems for the biographic, job performanceand attendance information, benefits systems for the benefitsinformation, and learning systems for the training and complianceinformation, in examples. At the same time, the ERM system can simply bea collection of local or remote databases that store the different typesof information associated with each employee.

The access control systems typically include access control readers.These readers are often installed at access points of the buildings tocontrol access to restricted areas, such as buildings or areas of thebuildings. Examples of access points include front and interior doors ofa building, elevators, hallways connecting two areas of a building, tolist a few examples. The access control readers authenticate identitiesof (or authorize) individuals and then permit those authenticatedindividuals to access the restricted areas through the access points.Typically, individuals interact with the access control readers byswiping keycards or bringing contactless smart cards within range(approximately 2-3 inches or 5 centimeters) of a reader. The accesscontrol readers read user information of the keycards, such ascredentials of the individuals, and then the access control systemsdetermine if the individuals are authorized to access the restrictedareas. If the individuals are authorized to enter the restricted areas,then the access control readers allow access to the restricted areas byunlocking locked doors, signaling that doors should be unlocked, orgenerating alarms upon unauthorized entry, for example.

More recently, frictionless access control systems are being proposedand designed. These systems typically rely on individuals carryingbeacon devices that can broadcast credentials, such as dedicated fobdevices or personal mobile computing devices such as tablet or smartphone computing devices. These systems are “frictionless” in that theindividual may not have made any overt gesture indicating a desire toaccess the restricted area, e.g., the individuals did not swipe akeycard. The access control systems will then monitor and track theindividuals as they move through the buildings and automatically openaccess points such as doors when approached, assuming that theindividuals are authorized to pass through those access points.

Enterprise surveillance systems are used to help protect people,property, and reduce crime. These systems are used to monitor buildings,lobbies, entries/exits, and secure areas within the buildings of theenterprises, to list a few examples. The surveillance systems alsoidentify illegal activity such as theft or trespassing, in examples. Atthe same time, these surveillance systems can also have business uses.They can track employee locations across different rooms withinbuildings and among the different buildings of the enterprises.

In these surveillance systems, surveillance cameras capture image dataof scenes. The image data is typically represented as two-dimensionalarrays of pixels. The cameras include the image data within streams, andusers of the system such as security personnel view the streams ondisplay devices such as video monitors. The image data is also typicallystored to a video management system (VMS) for later access and analysis.

SUMMARY OF THE INVENTION

It would be beneficial if the enterprises such as companies or stores orgovernmental entities could monitor the emotional state of individualsinteracting with them, especially as the individuals are entering orleaving, to determine whether the individuals pose potential threats orfor less critical uses such as simply whether employees or customers arehappy or not.

The proposed security system tracks individuals, obtains emotional stateinformation of the individuals, and determines whether the emotionalstate information suggests that employees or other individuals posepotential threats at the enterprise. The proposed system can alsoperform facial recognition to identify and track the individuals. Thesystem can then alert security personnel when the determined emotionalstates of the individuals indicate fear or anger, and can also restrictaccess to the building in response. In addition or alternatively, thesystem can be simply used to collect statistical information on theindividuals.

In general, according to one aspect, the invention features anenterprise employee monitoring system. The system might includesurveillance cameras and a facial recognition module, an emotionalanalyzer module and/or an employee database. The surveillance camerascapture image data including employee individuals within the enterprise.The facial recognition module identifies the individuals in the imagedata, and the emotional analyzer module determines an emotional state ofthe individuals based upon the image data. The employee database storesemployee information and the emotional state information from theemotional analyzer module, based upon the identification performed bythe facial recognition module.

The employee monitoring system can additionally generate statisticalanalysis of the emotional state of the individuals such as employees.Preferably, the statistical analysis is reported to an employee resourcemanagement system that maintains the employee database.

In one implementation, employee information is matched to theindividuals and/or to groups within the enterprise to determine grouplevel trends in the emotional states of the individuals. In examples,the groups can include institutional groups of the enterprise and socialgroups. Typically, the system reports the group level trends in theemotional states of the individuals to security guards and/or managementpersonnel and/or human resource managers.

In general, according to another aspect, the invention features anenterprise security system. The security system includes surveillancecameras that capture image data including the individuals within theenterprise, an emotion analyzer for determining an emotional state ofthe individuals based on the image data, and a security integrationsystem for signaling security guards based on the emotional statedetermined by the emotion analyzer module.

In general, according to another aspect, the invention features a methodof employee monitoring. The method includes capturing image dataincluding individuals within the enterprise, and identifying theindividuals in the image data. The method also includes determining anemotional state of the individuals based on the image data, and trackingthe emotional state information of the individuals and saving theinformation to an employee database of an employee resource managementsystem based on the identification of the individuals.

In general, according to yet another aspect, the invention features anenterprise security method. The method includes capturing image dataincluding the individuals within the enterprise, an emotion analyzermodule determining an emotional state of the individuals based on theimage data, and signaling security guards based on the emotional statedetermined by the emotion analyzer module.

The above and other features of the invention including various noveldetails of construction and combinations of parts, and other advantages,will now be more particularly described with reference to theaccompanying drawings and pointed out in the claims. It will beunderstood that the particular method and device embodying the inventionare shown by way of illustration and not as a limitation of theinvention. The principles and features of this invention may be employedin various and numerous embodiments without departing from the scope ofthe invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings, reference characters refer to the sameparts throughout the different views. The drawings are not necessarilyto scale; emphasis has instead been placed upon illustrating theprinciples of the invention. Of the drawings:

FIG. 1 is a schematic diagram showing an exemplary enterprise includingan enterprise security system according to the present invention;

FIG. 2 is a sequence diagram showing operation of the enterprisesecurity system when deployed as an employee monitoring system;

FIG. 3 is a sequence diagram showing an additional operation of themonitoring system, based upon emotional state information of theindividuals determined in FIG. 2; and

FIG. 4 is a sequence diagram showing yet another operation of themonitoring system, based upon emotional state information of theindividuals determined in FIG. 2.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention now will be described more fully hereinafter withreference to the accompanying drawings, in which illustrativeembodiments of the invention are shown. This invention may, however, beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein; rather, these embodiments areprovided so that this disclosure will be thorough and complete, and willfully convey the scope of the invention to those skilled in the art.

As used herein, the term “and/or” includes any and all combinations ofone or more of the associated listed items. Further, the singular formsand the articles “a”, “an” and “the” are intended to include the pluralforms as well, unless expressly stated otherwise. It will be furtherunderstood that the terms: includes, comprises, including and/orcomprising, when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof. Further, it will be understood that when anelement, including component or subsystem, is referred to and/or shownas being connected or coupled to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

FIG. 1 shows an exemplary enterprise including an enterprise securitysystem 100, which has been constructed according to the principles ofthe present invention.

The figure shows surveillance cameras 103, client computer systems 122,a security guard station 98, and various other computer systemsinstalled at a building 50 that carry out operations of the enterprise.The other computer systems include an ERM system 138, a VMS 110, and anaccess control system 128, which are interconnected via a local orenterprise network 210. Finally, a security integration system (SIS) 80provides some of the important functions of the present invention.

The cameras 103 capture image data 99 of the rooms 113 through theenterprise's building and thus images of individuals 60 in each room113. Cameras 103-1 and 103-2 are respectively installed in rooms 113-1and 113-2. The cameras 103 store their image data 99 to the VMS 110.Additionally, in the example of a retail store, additional cameras arepositioned to capture images at entrances to the building 50 in whichthe retail store is located.

The ERM system 138 is preferably as described hereinabove. As such, theERM system 138 has an employee database 139 that stores employee records123 of employees 60, and stores overall company status and trend metrics14. The employee records 123 include information for identifying eachemployee and locations of desks 45 within the building 50 for theemployees. In more detail, each employee record 123 typically includes aname 24, an employee number 32, a badge number 34, a badge photo 36,emotion information 142, an authorization level 151, and one or moredesk locations 161. The desk locations 161 list the locations of desksthat the employee is authorized to be present at or is otherwiseexpected to be near during work hours. In addition, the ERM system 138may also include other information such as databases that store the sameinformation for contractors and visitors to the enterprise.

In the case of the store, the ERM system's database 139 might furtherinclude records for customers.

The emotion information 142 includes stored emotional state informationfor each employee/customer 60. The emotional state information istime-stamped and collected over time.

The VMS 110 stores the image data 99 from the cameras 103 and includes acamera locations table 21, a facial recognition module 107, and anemotion analyzer module 127. The camera locations table 21 typically hasa record for each of the surveillance cameras 103. The record containssuch information as the room 113 in which the camera 103 is installed.It may also include information concerning the type of camera andpossibly even the field of view of the camera with respect to a map orfloor layout of the building 50.

The facial recognition module 107 determines facial recognitioninformation of the individuals captured in the image data and monitorsmovement and/or activity of individuals 60 within the rooms 113. Thefacial recognition module 107 determines facial recognition informationof the individuals captured in the image data and monitors movementand/or activity of individuals 60 within the rooms 113.

The emotion analyzer module 127 determines an emotional state for theindividuals /employees 60 from their images in the image data 99.Examples of these emotional states include anger, fear, and happiness,in examples. The emotional analyzer preferably also generates an emotionlevel, which could simply be a value between 1 and 10. For example, foreach individual in the image data, the analyzer module 127 generates astate, such as happy, and the level for that state. For example, “happy,5” for an individual would indicate an individual that was happier thanan individual with “happy, 3”.

The access control system 128 controls physical access to access points10 of the building 50. In the illustrated example, the access points aredoors, but may also include hallways or elevators or floors within thebuildings of the enterprise. Typically, the access control system 128further includes card readers for reading employee badges and/orfrictionless readers that might validate employees based on credentialsprovided by a mobile computing device such as a smart phone. In thisway, the access control system 128 is able to monitor movement ofindividuals through access points.

The VMS can be seeded with information concerning the enterprisesemployees and possibly customer or other individuals that interact withthe enterprise. For example, when the individuals 60 individuals areoriginally hired as employees, a security operator/guard or humanresources representative would create the employee record 123 for eachemployee in the employee database 139. The security guard also takes apicture of the employee's face to use as the badge photo 36, and usesthe facial recognition module 107 of the VMS 110 to create stored facialrecognition information for each of the employees. The security guardmight also create a baseline instance of emotional state information forthe employee, using the emotional analyzer module 127.

In the case of the store, the customers might go through a similarregistration process or the system would simply track repeat customersvia a customer number for example. Customers could be identified by nameor only using a customer number. Repeat customers and the emotionalstate of those customers would be tracked over the visits to the store.

The facial recognition information created and stored by the facialrecognition module 107 can be of different types. In one example, theinformation is a biometric identifier such as a facial signature of theindividual. In another example, the information is simply a still imageof the person's face extracted from the image data, also known as afacial patch.

The facial signature for an individual is a unique value or set ofvalues that represent the face of an individual/employee/customer. Thefacial recognition module 107 uses one or various predetermined facialsignature algorithms to create the facial signature, based upon variousfeatures of each person's face. These features include the eyes, nose,mouth, eyebrows, cheekbones, and chin of each face, and distancesbetween each of these features, in examples.

The facial recognition module also maps each instance of facialrecognition information (e.g. the facial signature or facial patch) foreach employee to a user credential or other identifier (OD). In thisway, the OD associated with each instance of stored facial recognitioninformation can be used to identify the individual for which the facialsignature was obtained.

The VMS 110 then stores the facial recognition information andassociated ID for identifying each employee or customer. In one example,the VMS stores this information locally to the VMS 110. In anotherexample, the VMS 110 might store this information to the employee orcustomer record 123 for each employee or customer.

The VMS also stores the obtained emotional state information to theemotion information 114 in the employee/customer record 123 for eachemployee/customer 60.

The enterprise security or monitoring system 100 generally operates asfollows.

The cameras 103 also capture and send the image data 99 of theindividuals to the VMS 110. The VMS 110 then stores the image data 99,performs facial recognition upon the image data 99 to identify theindividuals 60, and determines an emotional state of the individualsbased upon the image data 99.

From this analysis, the VMS 110 generates a meta data stream. The metadata stream includes an indication of each individual detected in theimage data from the various security cameras, a time stamp of the timeof detection, a location of the individual obtained by reference to thecamera locations table 21, an identification of the individual such asemployee or customer number, if the facial recognition module was ableto identify the individual, a detected emotional state of theindividual, and the determined level for that state.

The ERM system 138 receives surveillance meta data stream including thetime-stamped emotional state information sent from the VMS 110, andstores the information to the employee records 123 for each individualto the employee database 139 and/or a customer database as appropriate.

The security integration system (SIS) 80 functions to integrate theoperation of VMS 110 with the ERM system 138 and handle security issues.The SIS 80 can take many different forms. As such, the SIS 80 can be aseparate computer system or could be a process that executes on acomputer associated with the VMS 110 or the ER or even a separatecomputer system or a computer system integrated with the ERM computersystems.

The SIS 80 receives the surveillance meta data stream from the VMS 110,uses the IDs of the identified/tracked individuals in the stream toidentify the individuals, and can perform different operations basedupon the emotional state information associated with each individual inthe stream. In one example, if the emotional state information indicatesthat an individual is very upset or angry, the SIS 80 can signalsecurity guards to this effect. In another example, the SIS 80 obtainsthe associated employee records 123 for the individuals 60, and performsstatistical analysis on the information. In yet another example, the SIS80 can obtain employee records for other employees having the sameinstitutional information or social information as “upset” individuals.The SIS 80 can then determine trends in the emotional states of theemployees at a group level. Examples of these groups includeinstitutional groups (e.g. employees having the same manager, assignedproject, or building location) and social groups (e.g. employees havingthe same professional and/or professed political affiliations andoutside work interests).

FIG. 2 is a sequence diagram that illustrates a method of operation ofthe enterprise security monitoring system.

In step 202, the cameras 103-1 and 103-2 within the rooms 113-1 and113-2 of the building 50 capture the image data 99 of scenes in therooms 113. The cameras 103 send the image 99 to the VMS 110 for storageand subsequent analysis.

In step 204, the facial recognition module 107 of the VMS 110 locatesthe individuals 60 in the image data 99 and performs facial recognitionof the individuals to identify the individuals 60. For this purpose, thefacial recognition module 107 preferably uses the same facialrecognition algorithms used when the security guards first registeredthe individuals as employees.

According to step 206, the emotion analyzer module 127 obtainsinformation indicating an emotional state (e.g. upset/angry, calm, fear)and emotion level of each identified individual from the image data 99.In step 208, the VMS 110 obtains the location of each camera (e.g. roomnumber, entry door, exit door) from the camera locations table 21.

It can also be appreciated that the facial recognition module 107 andthe emotion analyzer module 127 can be included within and execute uponother components of the enterprise security monitoring system and/or theenterprise security system. In one example, the facial recognitionmodule 107 and the emotion analyzer module 127 might be integratedwithin the cameras 103 and execute upon a microcontroller of the cameras103. In other examples, these components might execute upon amicrocontroller or central processing unit (CPU) of the ACS 128 or belocated in a computer system that is remote to the enterprise, such as acloud system. In yet another example, the facial recognition module 107and the emotion analyzer module 127 could be located in differentcomponents.

In step 210, the VMS 110 sends a meta data stream to the ERM system 138,and to the SIS 80 in step 214. Specifically, for each employee and/orpossibly customer, the VMS 110 sends messages to update the emotioninformation 142 including state and level in the employee or customerrecord 123 for each identified employee/customer/individual in thestream. The messages/streams include the emotion information(state/level), the employee/customer number 32, badge number 34,associated location and time stamp, in examples. The ERM system 138 addsthe emotion information 142 to the employee/customer records 123 in step212.

The SIS 80 receives the meta data stream from the VMS in step 216. TheSIS then forwards messages for the upset employees and/or customersand/or other individuals to security guards and possibly the businessowner, to warn of potential threat posed by upset employees at theassociated locations, especially in the case where the levels of emotionare high, such as 9 or 10.

In step 218, the SIS 80 also performs statistical analysis to determinehow many employees are upset, and sends the statistical analysis to ERMsystem 138 in step 220. Then, in step 222, the ERM system 138 includesthe results of the statistical analysis in its overall company statusand trend metrics 14.

FIG. 3 shows additional operations that the SIS 80 can perform, basedupon the emotional state information received in FIG. 2 when functioningas an enterprise employee monitoring system.

In step 224, the SIS 80 requests the employee records 123 of the upsetemployee(s) previously determined in FIG. 2. The SIS 80 requests theemployee records from the ERM system 138 including the groups such asworking group or division or section of the enterprise, and the ERMsystem returns the records in step 226.

In step 228, the SIS 80 extracts the desk location (room and buildingnumber), manager name, formal and/or informal and/or social group, andassigned projects from the employee records 123 of each upset employee60.

Then, for each upset employee, the SIS 80 request employee records 123for other employees with matching desk locations, manager names, and/orassigned projects and/or other group, from the ERM system 138 in step230. The ERM system sends the matching employee records 123 in step 232.

According to step 234, the SIS 80 performs statistical analysis upon theemotion information 14 of the matching employee records to determinepossible trends at the group level. In this way, a security guard orhuman resources person might be able to infer that employees working ina particular building, for a particular manager, and/or on a particularproject are generally upset, in examples.

In step 236, the SIS 80 sends a message to alert security guards andpossibly business owners of the potential threat or problem posed by thegroups of employees. The SIS also sends the statistical analysis to ERMsystem 138 in step 238.

According to step 240, the ERM system 138 includes the results of thestatistical analysis in its overall company status and trend metrics 14.

FIG. 4 also shows additional operations that the SIS 80 can perform,based upon the employee emotional state information received in FIG. 2.

In step 242, the SIS 80 requests the employee records 123 of an “upset”employee 60 (i.e. an employee that the emotion analyzer module 127determined to have an upset or fearful emotional state). The ERM system138 sends the employee record 123 in step 244.

The SIS 80, in step 246, then perform statistical analysis upon theemotional information over a time period (e.g. weeks, months), for atime at which employee generally enters the building 50 each day, in oneexample. In this way, a consistently upset emotional state at this timefor the employee 60 could indicate a work-related issue that requiresfurther investigation by human resources personnel, in one example.

In another example, if an employee 60 is determined to be consistentlyin an upset emotional state when leaving work each day, then there maybe an issue outside of work affecting that employee 60. Such informationcan be used to determine potential threats posed by the employees 60before they manifest into actual threats, either in the workplace oroutside of the workplace, in examples.

In yet another example, if an employee 60 is determined to have a suddenchange in their emotional state as compared to their stored emotioninformation 14 over time, this could also be indicative of an employeeissue that requires follow-up by human resources personnel.

The above information can be used to determine potential threats posedby the employees 60 before they manifest into actual threats, either inthe workplace or outside of the workplace, in examples.

It can also be appreciated that cameras 103 located throughout thebuildings of an enterprise can capture the emotional states of theindividuals 60 as they work and interact with others. In this way, basedon the facial recognition information of manager and worker employeesand their emotional state information, the SIS 80 could compilestatistical information on interactions between groups of manager andworker employees over time to determine whether there are any personnelissues that require additional investigation by human resourcespersonnel.

While this invention has been particularly shown and described withreferences to preferred embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. An enterprise employee monitoring system,comprising: surveillance cameras that capture image data including theindividuals within the enterprise; a facial recognition module foridentifying the individuals in the image data; an emotion analyzermodule for determining an emotional state of the individuals based onthe image data; and a database for storing individual information andthe emotional state information from the emotion analyzer module basedon the identification performed by the facial recognition module.
 2. Asystem as claimed in claim 1, further comprising a security integrationsystem that generates statistical analysis of the emotional state of theindividuals.
 3. A system as claimed in claim 2, wherein the securityintegration system reports the statistical analysis to an employeeresource management system that maintains the employee database.
 4. Asystem as claimed in claim 1, wherein the security integration systemaccesses the employee information to match the individuals to groupswithin the enterprise to determine group level trends in the emotionalstates of the individuals.
 5. A system as claimed in claim 4, whereinthe groups include institutional groups of the enterprise.
 6. A systemas claimed in claim 4, wherein the groups include social groups.
 7. Asystem as claimed in claim 4, wherein the security integration systemreports the group level trends in the emotional states of theindividuals to security guards.
 8. An enterprise security system,comprising: surveillance cameras that capture image data including theindividuals within the enterprise; an emotion analyzer module fordetermining an emotional state of the individuals based on the imagedata; and a security integration system for signaling security guardsbased on the emotional state determined by the emotion analyzer module.9. A method of employee monitoring, comprising: capturing image dataincluding individuals within the enterprise; identifying the individualsin the image data; determining an emotional state of the individualsbased on the image data; and tracking the emotional state information ofthe individuals and saving the information to a database of an resourcemanagement system based on the identification of the individuals.
 10. Amethod as claimed in claim 9, further comprising generating statisticalanalysis of the emotional state of the individuals.
 11. A method asclaimed in claim 10, further comprising reporting the statisticalanalysis to an employee resource management system that maintains theemployee database.
 12. A method as claimed in claim 9, furthercomprising accessing the employee information to match the individualsto groups within the enterprise to determine group level trends in theemotional states of the individuals.
 13. A method as claimed in claim12, wherein the groups include institutional groups of the enterprise.14. A method as claimed in claim 12, wherein the groups include socialgroups.
 15. An enterprise security method, comprising: capturing imagedata including the individuals within the enterprise; an emotionanalyzer module determining an emotional state of the individuals basedon the image data; and signaling security guards based on the emotionalstate determined by the emotion analyzer module.
 16. A retail storemonitoring system, comprising: surveillance cameras that capture imagedata including the customers within the store; a facial recognitionmodule for identifying the customers in the image data; an emotionanalyzer module for determining an emotional state of the customersbased on the image data; and a customer database for storing customerinformation and the emotional state information from the emotionanalyzer module based on the identification performed by the facialrecognition module.